[ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries by Gasoonjia · Pull Request #18730 · pytorch/executorch

Gasoonjia · 2026-04-06T23:36:24Z

Stack from ghstack (oldest at bottom):

[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788
[ET Device Support] Extract shared device test utilities to reduce redundancy #18762
[ET Device Support] Add device tensor helper functions to TensorPtr API #18761
[ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies #18760
-> [ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730
[ET Device Support] Define et_copy runtime h2d and d2h copy ops #18729
[ET Device Support] Define AOT device copy ops registry #18728
[ET Device Support] CudaAllocator: device memory allocator for CUDA backend #18477
[ET Device Support] Module: allocate device memory for planned buffers #18476
[ET Device Support] MemoryManager: add per-buffer device metadata #18475
[ET Device Support] MethodMeta: expose per-buffer device placement API #18474
[ET Device Support] DeviceMemoryBuffer RAII class for device memory lifetime management #18473
[ET Device Support] Emitter reads non_const_buffer_device from graph meta #18472
[ET Device Support] Device-aware memory planning: separate buffers per device type #18375
[ET Device Support] Add NonConstBufferDevice schema for per-buffer device mapping #18330
[ET Device Support] DeviceAllocator interface and DeviceAllocatorRegistry #17535
[ET Device Support] Annotate device attributes of CUDA backend IO tensors cuda device #18080
[ET Device Support] Parse device info from serialized tensor in tensor_parser #18328
[ET Device Support] Propagate device info from TensorSpec into serialized Tensor #18079
[executorch] Propagate device metadata from partitioner result onto TensorSpecs #18078

Extend PropagateDevicePass to insert explicit et_copy._h2d_copy and
et_copy._d2h_copy ops at delegate boundaries, making the graph functional
by explicitly transferring data between CPU and device memory.

Key changes:

Inserts _h2d_copy before each delegate input, _d2h_copy after each output
Original input nodes stay CPU; h2d_copy output tagged as device
Getitem nodes inherit device; d2h_copy output tagged as CPU
Skip-copy optimizations via skip_h2d_for_method_inputs/skip_d2h_for_method_outputs
_parse_device_spec_value: lowercases string, raises ValueError for unknown types
_program.py passes config flags to PropagateDevicePass constructor

Differential Revision: D99636777

…elegate boundaries Extend PropagateDevicePass to insert explicit et_copy._h2d_copy and et_copy._d2h_copy ops at delegate boundaries, making the graph functional by explicitly transferring data between CPU and device memory. Key changes: - Inserts _h2d_copy before each delegate input, _d2h_copy after each output - Original input nodes stay CPU; h2d_copy output tagged as device - Getitem nodes inherit device; d2h_copy output tagged as CPU - Skip-copy optimizations via skip_h2d_for_method_inputs/skip_d2h_for_method_outputs - _parse_device_spec_value: lowercases string, raises ValueError for unknown types - _program.py passes config flags to PropagateDevicePass constructor Differential Revision: [D99636777](https://our.internmc.facebook.com/intern/diff/D99636777/) [ghstack-poisoned]

pytorch-bot · 2026-04-06T23:36:53Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18730

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 65 New Failures

As of commit 4b48ed3 with merge base b5ae0b9 ():

NEW FAILURES - The following jobs have failed:

Build Presets / apple (pybind) / build (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
Build Presets / linux (pybind, linux.2xlarge, executorch-ubuntu-22.04-clang12) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / windows (pybind) / build (gh)
Process completed with exit code 1.
periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 767c5e1959c7a6aeb373fca59f71a8e08eeed20cba44b7430b150e9f0dc370cd /exec failed with exit code 1
periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 9d244db216f1ba043156626b75b7085eceaff79c55cd9b4cc51764ec976fc68f /exec failed with exit code 1
periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 382441a8a1d1d20aee84db311c64932b68dd8a0d74005e779c1a67d4b4626a2a /exec failed with exit code 1
pull / android / build-android (gh)
Process completed with exit code 2.
pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux-android / linux-job (gh)
RuntimeError: Command docker exec -t fc0474a28f3828731dd1e3360c84879e37e67dd0f5114969ac62e9cf32a9a0e6 /exec failed with exit code 2
pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-lora-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-lora-multimethod-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-mcu-cortex-m-backend / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 55fb62db27a357fdf073b637ac19ac0d3376a5fd7055f20a80f44db8f53bc6c3 /exec failed with exit code 1
pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t f5e3561cc9fd64bd84a926849b8006c517cd8ca722bc16182ee4eff14cdb3194 /exec failed with exit code 1
pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 2c90e1a535495456c7b738c22eab234b0d99ebe0a40d8ea86769b0938d7cafc5 /exec failed with exit code 1
pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 4121f549062b8dd6ec7fc27e4630701569460d43e6950cacbc5989759f68737b /exec failed with exit code 1
pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t cfa95413314d13e2712a5d81fae42a823c82737bd25fc7a672bca74367c99746 /exec failed with exit code 1
pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 16286e16f2c1d4fa6bbace5efddbe860560c715b5f8e12814a987a5210ce6e3b /exec failed with exit code 1
pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 8ee881941a1d297cf192edf08d6e9dd7c53832ca53d66eba3f7bb4756654489b /exec failed with exit code 1
pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 8513077b7658ba3879b34b64fa69ae5ac9e3d693b08698e0ca3d3316b00941c9 /exec failed with exit code 1
pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t ed57bc83915a3664251c862e3da61951528c9e7b6531c0baf44c020859c760e4 /exec failed with exit code 1
pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 6529698d211a7e0aa42900796db076f274e455a62ca1a39f81bcda80eacc0b8d /exec failed with exit code 1
pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t cffd6ba4830f9597aa65ef736d8503a1b3ac6c7fa5b207f7f1b0a9891f73c7f6 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 865e77d2dedfd82ecf1a1d2a147f36f52e0f1acdd3e0ac9b6cf8bf1b32478d86 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t fd4cc6da015b46b97d27a8c7dc2c75ee12d0389beaaf7885f8556b5cd4eb91a7 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t 142c33237bb2cf29a53ced24119f1e33a219d5479fda833b77e6379315c2655c /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 659829f97aeca5e2000468b99fe81f669cc210d3f6917baaaad7e4460fc23e72 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 0371cc5b398f05010b1f92d2fb7107e356504c922dccd4f21f0324574b2bbafe /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t 091b2e0ac713ff692715865046527c6965284b2ab878d5dd75e70a32b5ebfcf9 /exec failed with exit code 1
pull / test-moshi-linux / linux-job (gh)
test_mimi
pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
RuntimeError: Command docker exec -t 1c62a85e41614d18eb355124ed6d6b27fd3fd8ab4f34fc4c6e27041054b8bec6 /exec failed with exit code 1
pull / test-openvino-linux / linux-job (gh)
RuntimeError: Command docker exec -t cb381eb884a8fa9f872b2e207fcf10d9bb5bd9e34b172822c8c8a3ec8ba44fac /exec failed with exit code 1
pull / test-parakeet-xnnpack-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-phi-3-mini-runner-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-vulkan-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t c4bf46485a5bacea039aa5f9d387d1017d907b425c437a0c09addad70fb58e95 /exec failed with exit code 1
pull / test-vulkan-operators-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / unittest / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 3bfa3b5a2e6fdffd3753b9cf7cfa89077511f7acba30b7beb42595646aa08c71 /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_no_target) / linux-job (gh)
RuntimeError: Command docker exec -t 8d3447d3de1abfd6ec338b590e12b4c09527ef08a385a2feb03b579930987191 /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t b6372e550abecc0946ba7b1f843cea5656246b5e1e241a4ed40ef01637921bc5 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-nxp-neutron / linux-job (gh)
RuntimeError: Command docker exec -t e1c7d0129bc20ef28ca707700f0eba07fc4c40a8f5cd995175c7b729351cfea5 /exec failed with exit code 2
pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
RuntimeError: Command docker exec -t c791bb52bc0ad71545b4298e9cf486e1ef30510b49bc7378fc341241da294fc8 /exec failed with exit code 2
pull / unittest-wasm-bindings / linux-job (gh)
RuntimeError: Command docker exec -t 6504a37b7880a1c224dc39d72e6e61aa8051991cc24c95d706f9ae700416e44b /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…elegate boundaries Extend PropagateDevicePass to insert explicit et_copy._h2d_copy and et_copy._d2h_copy ops at delegate boundaries, making the graph functional by explicitly transferring data between CPU and device memory. Key changes: - Inserts _h2d_copy before each delegate input, _d2h_copy after each output - Original input nodes stay CPU; h2d_copy output tagged as device - Getitem nodes inherit device; d2h_copy output tagged as CPU - Skip-copy optimizations via skip_h2d_for_method_inputs/skip_d2h_for_method_outputs - _parse_device_spec_value: lowercases string, raises ValueError for unknown types - _program.py passes config flags to PropagateDevicePass constructor Differential Revision: [D99636777](https://our.internmc.facebook.com/intern/diff/D99636777/) ghstack-source-id: 363318416 Pull Request resolved: #18730

github-actions · 2026-04-06T23:37:22Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…py ops at delegate boundaries" Extend PropagateDevicePass to insert explicit et_copy._h2d_copy and et_copy._d2h_copy ops at delegate boundaries, making the graph functional by explicitly transferring data between CPU and device memory. Key changes: - Inserts _h2d_copy before each delegate input, _d2h_copy after each output - Original input nodes stay CPU; h2d_copy output tagged as device - Getitem nodes inherit device; d2h_copy output tagged as CPU - Skip-copy optimizations via skip_h2d_for_method_inputs/skip_d2h_for_method_outputs - _parse_device_spec_value: lowercases string, raises ValueError for unknown types - _program.py passes config flags to PropagateDevicePass constructor Differential Revision: [D99636777](https://our.internmc.facebook.com/intern/diff/D99636777/) [ghstack-poisoned]

digantdesai

Review automatically exported from Phabricator review in Meta.

Gasoonjia requested review from JacobSzwejbka and larryliu0820 as code owners April 6, 2026 23:36

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 6, 2026

meta-codesync bot added fb-exported meta-exported labels Apr 6, 2026

Gasoonjia mentioned this pull request Apr 7, 2026

add device attribute in ETensor #17161

Open

digantdesai approved these changes Apr 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries#18730

[ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries#18730
Gasoonjia wants to merge 2 commits intogh/gasoonjia/160/basefrom
gh/gasoonjia/160/head

Gasoonjia commented Apr 6, 2026 •

edited

Loading

Uh oh!

pytorch-bot bot commented Apr 6, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 6, 2026

Uh oh!

digantdesai left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Gasoonjia commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18730

❌ 65 New Failures

Uh oh!

github-actions bot commented Apr 6, 2026

This PR needs a release notes: label

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Gasoonjia commented Apr 6, 2026 •

edited

Loading

pytorch-bot bot commented Apr 6, 2026 •

edited

Loading

This PR needs a `release notes:` label